* Added support for running checks on multiple tables ([#566](#566)). Added more flexibility and functionality in running data quality checks, allowing users to run checks on multiple tables in a single method call and as part of Workflows execution. Provided options to run checks for all configured run configs or for a specific run config, or for tables/views matching wildcard patterns. The CLI commands for running workflows have been updated to reflect and support these new functionalities. Additionally, new parameters have been added to configuration file to control the level of parallelism for these operations, such as `profiler_max_parallelism` and `quality_checker_max_parallelism`. A new demo has been added to showcases how to use the profiler and apply checks across multiple tables. The changes aim to improve scalability of DQX.
* Added New Row-level Checks: IPv6 Address Validation ([#578](#578)). DQX now includes 2 new row-level checks: validation of IPv6 address (`is_valid_ipv6_address` check function), and validation if IPv6 address is within provided CIDR block (`is_ipv6_address_in_cidr` check function).
* Added New Dataset-level Check: Schema Validation check ([#568](#568)). The `has_valid_schema` check function has been introduced to validate whether a DataFrame conforms to a specified schema, with results reported at the row level for consistency with other checks. This function can operate in non-strict mode, where it verifies the existence of expected columns with compatible types, or in strict mode, where it enforces an exact schema match, including column order and types. It accepts parameters such as the expected schema, which can be defined as a DDL string or a StructType object, and optional arguments to specify columns to validate and strict mode.
* Added New Row-level Checks: Spatial data validations ([#581](#581)). Specialized data validation checks for geospatial data have been introduced, enabling verification of valid latitude and longitude values, various geometry and geography types, such as points, linestrings, polygons, multipoints, multilinestrings, and multipolygons, as well as checks for Open Geospatial Consortium (OGC) validity, non-empty geometries, and specific dimensions or coordinate ranges. These checks are implemented as check functions, including `is_latitude`, `is_longitude`, `is_geometry`, `is_geography`, `is_point`, `is_linestring`, `is_polygon`, `is_multipoint`, `is_multilinestring`, `is_multipolygon`, `is_ogc_valid`, `is_non_empty_geometry`, `has_dimension`, `has_x_coordinate_between`, and `has_y_coordinate_between`. The addition of these geospatial data validation checks enhances the overall data quality capabilities, allowing for more accurate and reliable geospatial data processing and analysis. Running these checks requires Databricks serverless or cluster with runtime 17.1 or above.
* Added absolute and relative tolerance to comparison of datasets ([#574](#574)). The `compare_datasets` check has been enhanced with the introduction of absolute and relative tolerance parameters, enabling more flexible comparisons of decimal values. These tolerances can be applied to numeric columns.
* Added detailed telemetry ([#561](#561)). Telemetry has been enhanced across multiple functionalities to provide better visibility into DQX usage, including which features and checks are used most frequently. This will help us focus development efforts on the areas that matter most to our users.
* Allow installation in a custom folder ([#575](#575)). The installation process for the library has been enhanced to offer flexible installation options, allowing users to install the library in a custom workspace folder, in addition to the default user home directory or a global folder. When installing DQX as a workspace tool using the Databricks CLI, users are prompted to optionally specify a custom workspace path for the installation.  Allowing custom installation folder makes it possible to use DQX on [group assigned cluster](https://docs.databricks.com/aws/en/compute/group-access).
* Profile subset dataframe ([#589](#589)). The data profiling feature has been enhanced to allow users to profile and generate rules on a subset of the input data by introducing a filter option, which is a string SQL expression that can be used to filter the input data. This filter can be specified in the configuration file or when using the profiler, providing more flexibility in analyzing subsets of data. The profiler supports extensive configuration options to customize the profiling process, including sampling, limiting, and computing statistics on the sampled data. The new filter option enables users to generate more targeted and relevant rules, and it can be used to focus on particular segments of the data, such as rows that match certain conditions.
* Added custom exceptions ([#582](#582)). The codebase now utilizes custom exceptions to handle various error scenarios, providing more specific and informative error messages compared to generic exceptions.

BREAKING CHANGES!

* Workflows run by default for all run configs from configuration file. Previously, the default behaviour was to run them for a specific run config only.
* The following depreciated methods are removed from the `DQEngine`: `load_checks_from_local_file`, `load_checks_from_workspace_file`,  `load_checks_from_table`,  `load_checks_from_installation`, `save_checks_in_local_file`, `save_checks_in_workspace_file`, `save_checks_in_table`,, `save_checks_in_installation`, `load_run_config`.  For loading and saving checks, users are advised to use `load_checks` and `save_checks` of the `DQEngine` described [here](https://databrickslabs.github.io/dqx/docs/guide/quality_checks_storage/), which support various storage types.

* Added support for running checks on multiple tables
([#566](#566)). Added more
flexibility and functionality in running data quality checks, allowing
users to run checks on multiple tables in a single method call and as
part of Workflows execution. Provided options to run checks for all
configured run configs or for a specific run config, or for tables/views
matching wildcard patterns. The CLI commands for running workflows have
been updated to reflect and support these new functionalities.
Additionally, new parameters have been added to configuration file to
control the level of parallelism for these operations, such as
`profiler_max_parallelism` and `quality_checker_max_parallelism`. A new
demo has been added to showcases how to use the profiler and apply
checks across multiple tables. The changes aim to improve scalability of
DQX.
* Added New Row-level Checks: IPv6 Address Validation
([#578](#578)). DQX now
includes 2 new row-level checks: validation of IPv6 address
(`is_valid_ipv6_address` check function), and validation if IPv6 address
is within provided CIDR block (`is_ipv6_address_in_cidr` check
function).
* Added New Dataset-level Check: Schema Validation check
([#568](#568)). The
`has_valid_schema` check function has been introduced to validate
whether a DataFrame conforms to a specified schema, with results
reported at the row level for consistency with other checks. This
function can operate in non-strict mode, where it verifies the existence
of expected columns with compatible types, or in strict mode, where it
enforces an exact schema match, including column order and types. It
accepts parameters such as the expected schema, which can be defined as
a DDL string or a StructType object, and optional arguments to specify
columns to validate and strict mode.
* Added New Row-level Checks: Spatial data validations
([#581](#581)). Specialized
data validation checks for geospatial data have been introduced,
enabling verification of valid latitude and longitude values, various
geometry and geography types, such as points, linestrings, polygons,
multipoints, multilinestrings, and multipolygons, as well as checks for
Open Geospatial Consortium (OGC) validity, non-empty geometries, and
specific dimensions or coordinate ranges. These checks are implemented
as check functions, including `is_latitude`, `is_longitude`,
`is_geometry`, `is_geography`, `is_point`, `is_linestring`,
`is_polygon`, `is_multipoint`, `is_multilinestring`, `is_multipolygon`,
`is_ogc_valid`, `is_non_empty_geometry`, `has_dimension`,
`has_x_coordinate_between`, and `has_y_coordinate_between`. The addition
of these geospatial data validation checks enhances the overall data
quality capabilities, allowing for more accurate and reliable geospatial
data processing and analysis. Running these checks requires Databricks
serverless or cluster with runtime 17.1 or above.
* Added absolute and relative tolerance to comparison of datasets
([#574](#574)). The
`compare_datasets` check has been enhanced with the introduction of
absolute and relative tolerance parameters, enabling more flexible
comparisons of decimal values. These tolerances can be applied to
numeric columns.
* Added detailed telemetry
([#561](#561)). Telemetry
has been enhanced across multiple functionalities to provide better
visibility into DQX usage, including which features and checks are used
most frequently. This will help us focus development efforts on the
areas that matter most to our users.
* Allow installation in a custom folder
([#575](#575)). The
installation process for the library has been enhanced to offer flexible
installation options, allowing users to install the library in a custom
workspace folder, in addition to the default user home directory or a
global folder. When installing DQX as a workspace tool using the
Databricks CLI, users are prompted to optionally specify a custom
workspace path for the installation. Allowing custom installation folder
makes it possible to use DQX on [group assigned
cluster](https://docs.databricks.com/aws/en/compute/group-access).
* Profile subset dataframe
([#589](#589)). The data
profiling feature has been enhanced to allow users to profile and
generate rules on a subset of the input data by introducing a filter
option, which is a string SQL expression that can be used to filter the
input data. This filter can be specified in the configuration file or
when using the profiler, providing more flexibility in analyzing subsets
of data. The profiler supports extensive configuration options to
customize the profiling process, including sampling, limiting, and
computing statistics on the sampled data. The new filter option enables
users to generate more targeted and relevant rules, and it can be used
to focus on particular segments of the data, such as rows that match
certain conditions.
* Added custom exceptions
([#582](#582)). The codebase
now utilizes custom exceptions to handle various error scenarios,
providing more specific and informative error messages compared to
generic exceptions.

BREAKING CHANGES!

* Workflows run by default for all run configs from configuration file.
Previously, the default behaviour was to run them for a specific run
config only.
* The following depreciated methods are removed from the `DQEngine`:
`load_checks_from_local_file`, `load_checks_from_workspace_file`,
`load_checks_from_table`, `load_checks_from_installation`,
`save_checks_in_local_file`, `save_checks_in_workspace_file`,
`save_checks_in_table`,, `save_checks_in_installation`,
`load_run_config`. For loading and saving checks, users are advised to
use `load_checks` and `save_checks` of the `DQEngine` described
[here](https://databrickslabs.github.io/dqx/docs/guide/quality_checks_storage/),
which support various storage types.